Nearest Neighbor Classi cation from Multiple Feature Subsets

Combining multiple classi ers is an e ective technique for improving accuracy. There are many general combining algorithms, such as Bagging, Boosting, or Error Correcting Output Coding, that signi cantly improve classi ers like decision trees, rule learners, or neural networks. Unfortunately, these combining methods do not improve the nearest neighbor classi er. In this paper, we present MFS, a combining algorithm designed to improve the accuracy of the nearest neighbor (NN) classi er. MFS combines multiple NN classi ers each using only a random subset of features. The experimental results are encouraging: On 25 datasets from the UCI Repository, MFS signi cantly outperformed several standard NN variants and was competitive with boosted decision trees. In additional experiments, we show that MFS is robust to irrelevant features, and is able to reduce both bias and variance components of error.